Image processing apparatus and image processing method

ABSTRACT

An apparatus for identifying a candidate area in a first image corresponding to an object in a second image, includes a memory and a processor to divide the plurality of candidate areas into a plurality of small candidate areas, divide an image area of the object into a plurality of small areas, perform first comparison processing for a first part, when there is a first candidate area lacking image information of the small candidate area corresponding to the first part, perform second comparison processing for a second part, predict missing result on the small candidate area corresponding to the first part in the first candidate area based on a result of the first comparison processing on a candidate area other than the first candidate area, and a result of the second comparison processing on the plurality of candidate areas, and identify the candidate area based on a prediction.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority of theprior Japanese Patent Application No. 2015-032765, filed on Feb. 23,2015, the entire contents of which are incorporated herein by reference.

FIELD

The embodiments discussed herein are related to image processing.

BACKGROUND

In person identification as an anticrime measure, when a newly capturedimage is matched with an image captured in advance, processing forassociating the same person in the two images is sometimes performed.Also, in a flow line analysis of a person, processing for associatingthe same person in the two images captured by two cameras is sometimesperformed. In such processing, a still image or a video is used.

In a flow line analysis, association is carried out using images of theentire body of a person, and various parts of the body is used asfeatures. In this case, the features of individual parts are extractedfrom the two images and used for comparison, and it is possible toconsider a most similar person as the same person based on thesimilarity degrees of the features.

A pattern identification method for identifying an input pattern of animage, or the like using a binary tree table is disclosed in JapaneseLaid-open Patent Publication No. 4-112391, for example. In this patternidentification method, a feature vector is generated from the inputpattern, and pattern identification is performed by referencing a binarytree table regarding the pattern predicted as a candidate categoryprovided in advance from this feature vector.

An image segmentation method for segmenting an image of a human bodyinto parts is disclosed, for example, in D. Qin, Q. Jianzhong, L. Fang,S. Xiangbin, D. Qin and Y. Hongping, “A Human Body Part SegmentationMethod Based on Markov Random Field,” International Conference onControl Engineering and Communication Technology (ICCECT), 2012.

SUMMARY

According to an aspect of the invention, an image processing apparatusfor identifying a candidate area, from among a plurality of candidateareas in a first image, corresponding to an object in a second image,includes a memory and a processor coupled to the memory and configuredto divide the plurality of candidate areas in the first image into aplurality of small candidate areas, respectively, divide an image areaof the object in the second image into a plurality of small areas,perform, regarding each of the plurality of candidate areas, firstcomparison processing for comparing a small area corresponding to afirst part with a small candidate area corresponding to the first part,the first part being a part of the object, when there is no a firstcandidate area lacking image information of the small candidate areacorresponding to the first part, identify the candidate areacorresponding to the object based on a result of the first comparisonprocessing, and when there is the first candidate area, perform secondcomparison processing for comparing another small area corresponding toa second part with another small area corresponding to the second partfor each of the plurality of candidate areas, the second part beingdifferent from the first part, predict missing result on the smallcandidate area corresponding to the first part in the first candidatearea based on the result of the first comparison processing on acandidate area other than the first candidate area, and a result of thesecond comparison processing on the plurality of candidate areas, andidentify the candidate area corresponding to the object based on theresult of the first comparison processing and a result of a prediction.

The object and advantages of the invention will be realized and attainedby means of the elements and combinations particularly pointed out inthe claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory and arenot restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a functional configuration diagram of an image processingapparatus;

FIG. 2 is a flowchart of image processing;

FIG. 3 is a functional configuration diagram illustrating a specificexample of the image processing apparatus;

FIG. 4 is a diagram illustrating two images captured by two cameras;

FIG. 5 is a diagram illustrating an example of areas corresponding toindividual parts of a body;

FIG. 6 is a diagram illustrating feature information;

FIG. 7 is a diagram illustrating a person image from which the featurequantity of a face is missing;

FIG. 8 is a flowchart of certainty factor calculation processing;

FIG. 9 is a diagram illustrating a sample of person images captured bytwo cameras;

FIG. 10 is a diagram illustrating certainty factor information;

FIG. 11 is a flowchart illustrating a specific example of imageprocessing;

FIG. 12 is a diagram illustrating a similarity degree of each part; and

FIG. 13 is a hardware configuration diagram of an information processingapparatus.

DESCRIPTION OF EMBODIMENTS

The related-art image processing has the following problems.

In a flow line analysis, or the like, feature extraction from an imagesometimes fails depending on the installation condition of a camera. Inparticular, when the optical axis of a camera is close to horizontal,for example, a person is blocked by another person or an object so thatan image of a partial part of the person is sometimes not captured. Insuch a case, the features allowed to be used for comparing the twoimages are restricted, and thus it becomes difficult to correctlyassociate the two images.

In this regard, such a problem arises not only when personidentification or a flow line analysis is performed in order to take ananticrime measure, but also when the same person is associated betweentwo images in the other image processing.

According to an embodiment of the present disclosure, it is desirable tocompare persons in images with high precision even if a part of an areain an image is not suitable for calculating a similarity degree.

In the following, a detailed description will be given of the embodimentwith reference to the drawings.

FIG. 1 illustrates an example of a functional configuration of an imageprocessing apparatus according to the embodiment. An image processingapparatus 101 includes a storage unit 111, a comparison unit 112, and anoutput unit 113.

The storage unit 111 stores a person image 121, and a plurality ofobject images corresponding to a plurality of persons, respectively. Theperson image 121 includes a first area and a second area. An objectimage 131 out of the plurality of object images includes a third areacorresponding to the first area, and a fourth area corresponding to thesecond area. An object image 132 out of the plurality of object imagesincludes a fifth area corresponding to the first area, and a sixth areacorresponding to the second area.

The comparison unit 112 generates a comparison result by comparing theperson image 121 with the plurality of object images, and the outputunit 113 outputs the comparison result.

FIG. 2 is a flowchart illustrating an example of image processingperformed by the image processing apparatus 101 in FIG. 1. If thefeature quantity of the third area is not suitable for calculating asimilarity degree, the comparison unit 112 obtains a fourth similaritydegree between the feature quantity of the first area and the featurequantity of the third area based on a first similarity degree, a secondsimilarity degree, and a third similarity degree (step 201). The firstsimilarity degree is a similarity degree between the feature quantity ofthe first area and the feature quantity of the fifth area, the secondsimilarity degree is a similarity degree between the feature quantity ofthe second area and the feature quantity of the fourth area, and thethird similarity degree is a similarity degree between the featurequantity of the second area and the feature quantity of the sixth area.

Then the comparison unit 112 generates a comparison result by comparingthe person image with the plurality of object images based on the firstsimilarity degree and the fourth similarity degree (step 202), and theoutput unit 113 outputs the comparison result (step 203).

With the image processing apparatus 101 in FIG. 1, it is possible tocompares persons between images with high precision even if a part of anarea in the images is not suitable for similarity degree calculation.

FIG. 3 illustrates a specific example of the image processing apparatus101 in FIG. 1. The image processing apparatus 101 in FIG. 3 includes astorage unit 111, a comparison unit 112, an output unit 113, a certaintyfactor calculation unit 301, a part identification unit 302, and afeature extraction unit 303.

The storage unit 111 stores the person image 121, and N pieces of objectimage 311-1 to object image 311-N corresponding to N persons (N is aninteger of two or more), respectively. For example, the person image 121is a person image included in an image captured by a first camera, andthe object image 311-1 to object image 311-N are person images includedin an image captured by a second camera.

FIG. 4 illustrates an example of two images captured by two cameras. Animage 401 captured by a first camera includes a person A, and an imagecaptured by 402 captured by a second camera includes a person A, aperson B, a person C, and a person D. In this case, A1 is given to theimage of the person A in the image 401 as identification information(ID), and A2, B2, C2, and D2 are given to the images of the person A,the person B, the person C, and the person D, respectively, in the image402 as ID.

The image processing apparatus 101 compares the image A1 with each ofthe image A2, the image B2, the image C2, and the image D2, andidentifies a person in the image 402 corresponding to the image A1. Theimage A1 extracted from the image 401 is stored in the storage unit 111as the person image 121, and the image A2, the image B2, the image C2,and the image D2 that are extracted from the image 402 are stored in thestorage unit 111 as object image 311-1 to object image 311-4. In thiscase, N=4.

The part identification unit 302 identifies an area corresponding toeach part of a body from each of the person image 121 and the objectimage 311-1 to the object image 311-N, and divides each of the imagesinto a plurality of the identified areas. It is possible for the partidentification unit 302 to divide each of the images into a plurality ofareas corresponding to a plurality of parts, respectively, using theimage segmentation method of the following non-patent literature, forexample. D. Qin, Q. Jianzhong, L. Fang, S. Xiangbin, D. Qin and Y.Hongping, “A Human Body Part Segmentation Method Based on Markov RandomField,” International Conference on Control Engineering andCommunication Technology (ICCECT), 2012.

FIG. 5 illustrates an example of areas corresponding to individual partsof a body. The part identification unit 302 generates a map image 502 inwhich five parts, namely, hair, a face, an upper body, a lower body, andshoes are distinguished from a person image 501, and divides the mapimage 502 into an area 511 to an area 516. The area 511 corresponds tohair, the area 512 corresponds to a face, the area 513 corresponds to anupper body, the area 514 corresponds to a lower body, the area 515corresponds to shoes, and the area 516 corresponds to a backgroundportion.

The part identification unit 302 records a position and size informationof an area of each part obtained from the person image 121 in thefeature information 312-0. Also, the part identification unit 302records a position and size information of an area of each part obtainedfrom the object images 311-1 to the object image 311-N in acorresponding one of the feature information 312-1 to the featureinformation 312-N.

FIG. 6 illustrates an example of the feature information 312-0 to thefeature information 312-N. The feature information in FIG. 6 correspondsto the map image 502 in FIG. 5, start point coordinates (sx, sy)represent coordinates of an upper left vertex of an area correspondingto a part, and end point coordinates (ex, ey) represent coordinates of alower right vertex of the area.

In this example, the coordinate origin is the upper left vertex of themap image 502, and the width and the height of the map image 502 are 50and 100, respectively. For example, the coordinates of a start point P1of the hair area 511 are (20, 10), and the coordinates of an end pointP2 are (40, 15). It is possible to express the position and the size ofthe area of each part by start point coordinates (sx, sy) and end pointcoordinates (ex, ey).

The feature extraction unit 303 extracts feature quantities from areasof individual parts in the map images generated from the person image121 and the object image 311-1 to the object image 311-N, respectively,and records the feature quantities in the feature information 312-0 tothe feature information 312-N, respectively. For the feature quantities,it is possible to use information on a color of an area, a colordifference signal, a gray level, a brightness, a shape, a texture, anedge, a contour, and the like. For the contour information, Histogramsof Oriented Gradients (HOG) may be used, for example.

In the feature information in FIG. 6, the average value of theindividual colors (R, G, B) in an area of each part is extracted as afeature quantity. For example, the average value of the color of thearea 511 of the hair is (10, 11, 10). In place of the average value inRGB, the other statistical values, such as the median, the maximumvalue, the minimum value, or the like may be used.

Also, it is possible to record a plurality of colors as a featurequantity of the area of each part. In this case, it is possible for thefeature extraction unit 303 to divide the color of an area into aplurality of part areas based on the similarity of color, to determine apredetermined number of part areas in descending order of the size ofthe area, and to record the statistical value of the color of each partarea as a feature quantity.

The certainty factor calculation unit 301 calculates a certainty factorfor each part of the body, and generates certainty factor information313 indicating certainty factors of a plurality of parts. The certaintyfactor of each part is an index indicating the certainty of thecomparison result of the comparison between the two person images basedon the feature quantities of the area of the part. The higher thecertainty factor, the higher the possibility of correctness of thecomparison result, and the lower the certainty factor, the lower thepossibility of correctness of the comparison result.

The comparison unit 112 compares the person image 121 and the objectimage 311-1 to object image 311-N using the feature information 312-0 tothe feature information 312-N and the certainty factor information 313,and identifies an object image corresponding to the person image 121.Then the comparison unit 112 generates a comparison result 314indicating the identified object image, and the output unit 113 outputsthe comparison result 314.

At this time, the comparison unit 112 calculates the similarity degreebetween a feature quantity of a part having a large certainty factoramong the feature quantities of a plurality of parts included in thefeature information 312-0, and a feature quantity of the same partincluded in the feature information 312-i (i=1 to N), for example. Thenit is possible for the comparison unit 112 to identify the object image311-i corresponding to the maximum value of the similarity degree as anobject image corresponding to the person image 121.

Incidentally, it is possible for the part identification unit 302 togenerate the map image 502 in which the parts are distinguished based onthe feature quantities of the individual parts included in the personimage 501 in FIG. 5. For example, if a color range (upper limit valueand lower limit value) for each part is given, it is possible toidentify an area of a corresponding part by connecting adjacent pixelshaving the given range of color with each other so as to identify thearea of the corresponding part. Then in the map image 502, thestatistical value (the average value, or the like) of the color of thepixels are given to all the pixels of each part as a feature quantity.

However, a partial part is sometimes not included in a person imagedepending on a photographing environment, such as setting conditions ofa camera, and the like. For example, if a person is blocked by anotherperson or an object so that a specific part is hidden, that part is notincluded in the person image. In this case, an image of the other personor the object is included in an area in which an image of the part oughtto be originally included, and the feature quantity of that area is notsuitable for calculating the similarity degree.

Also, if a picture of a person looking downward is taken, the face ofthe person is sometimes not included in the person image. In this case,the image of a head top part is included in an area where the face ofthe person ought to be included, and thus the feature quantity of thatarea is not suitable for calculating the similarity degree of the face.

In this manner, if the feature quantity of an area corresponding to acertain part in a person image is not suitable for calculating thesimilarity degree, a pixel having the color range given to the part isnot detected, and thus the feature quantity of that part is missing fromthe map image.

FIG. 7 illustrates an example of a person image from which the featurequantity of a face is missing. In a person image 701, if a part of aface of a person is hidden by some kind of obstacle, it becomesdifficult to identify the face part, and thus the feature quantity ofthe face is missing from a map image 702. The map image 702 includes anarea 711 corresponding to hair, an area 712 corresponding to an upperbody, an area 713 corresponding to a lower body, and an area 714corresponding to shoes, but does not include an area corresponding tothe face. In this case, the feature extraction unit 303 records values(−1, −1, −1) indicating that the feature quantity is missing in place ofthe feature quantity of the face in the feature information in FIG. 6.

If the feature quantity of a partial part is missing from any one of thefeature information 312-1 to feature information 312-N, the comparisonunit 112 obtains an alternative value of the similarity degree of themissing part based on the certainty factor of each part. Then thecomparison unit 112 uses the obtained alternative value as thesimilarity degree of the part, and compares the person image 121 withthe object image 311-1 to the object image 311-N.

In parts of a person's body, there are parts having a high possibilityof allowing to identify whether the same person or not, and parts havinga low possibility. For example, among three parts, namely a face, anupper body, and shoes, when the similarity degree of the featurequantity of the face is high, there is a high possibility of the sameperson. On the other hand, when the similarity degree of the featurequantity of the shoes is high, even if the similarity degree isconsiderably high, the possibility of the same person becomes lowcompared with the case where the similarity degree of the featurequantity of a face is high. Accordingly, it is thought that thecertainty factor of a face is higher than the certainty factor of shoes.

Between two parts having a high certainty factor, there is little chancethat the magnitude relation of the similarity degrees indicated by thefeature quantities of a plurality of object images changes compared witha part having a low certainty factor. For example, when the certaintyfactors of the face and the upper body are relatively high, if thesimilarity degree of the face of a person X is higher in descendingorder of the person A, the person B, and the person C, it often happensthat the similarity degree of the upper body of the person X is higherin descending order of the person A, the person B, and the person C.

Here, consider the case where the feature quantity of the face of theperson B and the feature quantities of the upper bodies of the person Aand the person B are extracted, but the feature quantity of the face ofthe person A is missing. In this case, if the similarity degree of theupper body of the person A is higher than the similarity degree of theupper body of the person B, it is expected that the similarity degree ofthe face of the person A is higher than the similarity degree of theface of the person B. Thus, it is preferable to set a higher value thanthe similarity degree of the face of the person B as an alternativevalue of the similarity degree of the face of the person A.

In this manner, even if the feature quantity of the area correspondingto a certain part in the object image 311-i is not suitable forcalculating the similarity degree, it becomes possible to compare theperson image 121 and the object image 311-i with high precision bysetting a suitable alternative value as the similarity degree of thepart. Thereby, it is possible to identify an object image that is mostsimilar to the person image 121 out of the object image 311-1 to theobject image 311-N.

FIG. 8 is a flowchart illustrating certainty factor calculationprocessing performed by the certainty factor calculation unit 301. Asillustrated in FIG. 9, in the certainty factor calculation processing, acertainty factor of each part of a body is calculated using a personimage group 901 including M persons (M is an integer of 2 or more)captured by the first camera, and a person image group 902 including thesame M persons captured by the second camera as a sample. The personimage group 901 includes a person image 911-1 to a person image 911-M,and the person image group 902 includes a person image 912-1 to a personimage 912-M. It is desirable that the sample number M is a sufficientlylarge integer.

The persons in the person image 911-1 to the person image 911-M and theperson image 912-1 to the person image 912-M are individually known, andthe storage unit 111 stores information indicating correct associationbetween the person images captured by the two cameras with each other inadvance. Further, the storage unit 111 stores the feature quantities ofthe individual parts extracted from the individual person images.

First, the certainty factor calculation unit 301 selects a person imageto be processed from the person image group 901 (step 801). Then thecertainty factor calculation unit 301 calculates the similarity degreeof a feature quantity of each part between the selected person image andeach person image in the person image group 902, and identifies a personimage in the person image group 902, which corresponds to the maximumvalue of the similarity degree for each part (step 802).

For example, when the feature quantity of a face is used, the certaintyfactor calculation unit 301 calculates the similarity degrees betweenthe feature quantity of the face of the selected person image, and thefeature quantity of the face of each person image in the person imagegroup 902, and associates a person image in the person image group 902,which corresponds to the maximum value of the similarity degree, withthe selected person image. Regarding the parts, such as hair, an upperbody, a lower body, shoes, and the like, it is possible to identify aperson image in the person image group 902 that corresponds to theselected person image in the same manner. The person images associatedin this manner are sometimes different for each part.

Next, the certainty factor calculation unit 301 checks whether all theperson images in the person image group 901 have been selected or not(step 803), and if there is an unselected person image (step 803, NO),the certainty factor calculation unit 301 repeats the processing of step801 and after that for the next person image.

If all the person images in the person image group 901 have beenselected (step 803, YES), the certainty factor calculation unit 301compares an association result of each person image for each part with acorrect association, and obtains a certainty factor of each part (step804). At this time, it is possible for the certainty factor calculationunit 301 to obtain a ratio of the number of person images having acorrect association to M, which is the number of person images in theperson image group 901, as a certainty factor.

For example, if 80 person images out of 100 person images in the personimage group 901 are associated with the corresponding correct personimages in the person image group 902 using the feature quantities of theface, the certainty factor of the face becomes 80%. For the other parts,it is possible to obtain ratios of the person images having beenassociated correctly as certainty factors in the same manner.

Next, the certainty factor calculation unit 301 determines a priority ofeach part at the time of comparing the person image 121 with the objectimage 311-1 to the object image 311-N in accordance with the certaintyfactor of each part (step 805). At this time, it is possible for thecertainty factor calculation unit 301 to determine priorities indescending order of the size of the certainty factors. Then thecertainty factor calculation unit 301 generates the certainty factorinformation 313 indicating the certainty factor and the priority of eachpart, and stores the certainty factor information 313 in the storageunit 111.

FIG. 10 illustrates an example of the certainty factor information 313regarding three parts, namely a face, shoes, and hair. In this example,the certainty factors for a face, shoes, and hair are 80%, 60%, and 40%,respectively, and the priorities of a face, shoes, and hair are thefirst, the second, and the third, respectively.

FIG. 11 is a flowchart illustrating a specific example of imageprocessing performed by the image processing apparatus 101 in FIG. 3after the certainty factor information 313 is generated. First, the partidentification unit 302 identifies an area corresponding to each part ofa body from each of the person image 121 and the object image 311-1 tothe object image 311-N, and generates a map image segmented into aplurality of areas (step 1101).

Next, the feature extraction unit 303 extracts a feature quantity froman area of each part in the map image generated from each of the personimage 121 and the object image 311-1 to the object image 311-N, andgenerates the feature information 312-0 to the feature information 312-N(step 1102).

Next, the comparison unit 112 calculates a similarity degree of thefeature quantity of each part between the feature information 312-0 andthe feature information 312-1 to the feature information 312-N (step1103). Here, if the feature quantity of any one of the parts of thefeature information 312-i is missing, the calculation of the similaritydegree of the part is omitted.

FIG. 12 illustrates an example of the similarity degrees of the face,the shoes, and the hair that are calculated between the image A1 and theimage A2 to the image D2 in FIG. 4. In this example, the featurequantities of the face and the hair in the image A2 are missing, andthus the similarity degrees of the face and the hair in the image A2 aremissing. In the same manner, the feature quantity of the shoes in theimage D2 is missing, and thus the similarity degree of the shoes in theimage D2 is missing. The similarity degrees in any parts are not higherthan 100.

Next, the comparison unit 112 compares the similarity degree of thefirst part among the similarity degrees of the individual partscalculated individually from the feature information 312-1 to thefeature information 312-N in accordance with the priorities indicated inthe certainty factor information 313 (step 1104). Then the comparisonunit 112 checks whether the similarity degree regarding the part ismissing or not (step 1105).

If the similarity degree is not missing (step 1105, NO), the comparisonunit 112 identifies an object image corresponding to the person image121 based on the similarity degree of the first part (step 1108). Inthis case, the comparison unit 112 compares the similarity degrees ofthe first part among the image A2 to the image D2, and identifies animage corresponding to the maximum value of the similarity degree as theobject image corresponding to the image A1. Then the comparison unit 112generates the comparison result 314 indicating the person of theidentified person image, and the output unit 113 outputs the comparisonresult 314.

On the other hand, if the similarity degree is missing (step 1105, YES),the comparison unit 112 obtains an alternative value of the similaritydegree of the missing part based on the magnitude relation of thesimilarity degree of the second part or less (step 1106). For themagnitude relation of the similarity degree, it is desirable to use themagnitude relation of the similarity degree of a part having a priorityas close to the first as possible.

For example, if the first part is a face, and the second part is shoes,an alternative value of the similarity degree of the face is set suchthat the magnitude relation of the similarity degree of the face matchesthe magnitude relation of the similarity degree of the shoes among theimage A2 to the image D2.

Then the comparison unit 112 compares the similarity degrees of theobject image 311-1 to the object image 311-N using the obtainedalternative value as the similarity degree of the first part (step1107), and performs the processing in step 1108.

For example, if the certainty factor information in FIG. 10 is used, thepriorities of a face, shoes, and hair are the first, the second, and thethird, respectively. Thus, the comparison unit 112 compares thesimilarity degrees of the face having the first priority among the imageA2 to the image D2 in FIG. 12. Here, it is understood that thesimilarity degree of the face in the image A2 is missing, and thesimilarity degree 80 of the face in the image C2 is the maximum valueamong the remaining image B2 to image D2. Accordingly, the candidateimage corresponding to the image A1 is narrowed down to the image A2 andthe image C2.

The similarity degree of the face in the image A2 is missing, and thusthe comparison unit 112 compares the similarity degrees of the shoeshaving the second priority. Here, it is understood that although thesimilarity degree of the shoes in the image D2 is missing, thesimilarity degrees of the shoes of the image A2 and the image C2 are notmissing. Thus, the comparison unit 112 obtains an alternative value ofthe similarity degree of the face in the image A2 by the followingexpression using the similarity degree 80 of the face in the image C2,and a ratio 50/10 of the similarity degree 50 of the shoes in the imageA2 to the similarity degree 10 of the shoes in the image C2.80×(50/10)=400  (1)

Here, if it is assumed that the upper limit value of the similaritydegree is 100, the comparison unit 112 corrects the obtained alternativevalue 400 to the upper limit value of 100, and sets the alternativevalue 100 in the similarity degree of the face in the image A2. As aresult, the similarity degree of the face in the image A2 becomes higherthan the similarity degree of the face in the image C2, and thus theimage corresponding to the image A1 is determined to be the image A2.

It is possible for the comparison unit 112 to further obtain analternative value of the similarity degree of the face in the image A2using the similarity degree of the face and the shoes in the image B2.In this case, the comparison unit 112 obtains the average value of thesimilarity degree 10 of the shoes in the image B2 and the similaritydegree 10 of the shoes in the image C2 by the following expression.(10+10)/2=10  (2)

Also, the comparison unit 112 obtains the average value of thesimilarity degree 65 of the face in the image B2 and the similaritydegree 80 of the face in the image C2 by the following expression.(65+80)/2=72.5  (3)

Then the comparison unit 112 obtains an alternative value of thesimilarity degree of the face in the image A2 by the followingexpression using the average value 72.5 in Expression (3), and the ratio50/10 of the similarity degree 50 of the shoes in the image A2 to theaverage value 10 in Expression (2).72.5×(50/10)=362.5  (4)

The comparison unit 112 modifies the obtained alternative value 362.5 tothe upper limit value of 100, and sets the alternative value 100 in thesimilarity degree of the face in the image A2.

It is also possible for the comparison unit 112 to further obtain analternative value of the similarity degree of the face in the image A2using the certainty factors of the face and the shoes. In this case, thecomparison unit 112 reflects the certainty factors on the similaritydegrees of the shoes in the image A2 to the image C2 by the followingexpression.A2: 50×60%=30  (5)B2: 10×60%=6  (6)C2: 10×60%=6  (7)

Next, the comparison unit 112 obtains the average value of thesimilarity degree 10 in Expression (6) and the similarity degree 10 inExpression (7) by the following expression.(6+6)/2=6  (8)

Also, the comparison unit 112 reflects the certainty factor on thesimilarity degrees of the faces in the image B2 and the image C2 by thefollowing expression.B2: 65×80%=52  (9)C2: 80×80%=64  (10)

Next, the comparison unit 112 obtains the average value of thesimilarity degree 52 in Expression (9) and the similarity degree 64 inExpression (10) by the following expression.(52+64)/2=58  (11)

Then the comparison unit 112 obtains an alternative value of thesimilarity degree of the face in the image A2 by the followingexpression using the average value 58 in Expression (11), and the ratio30/6 of the similarity degree 30 in Expression (5) to the average value6 in Expression (8).58×(30/6)=290  (12)

The comparison unit 112 modifies the obtained alternative value 290 tothe upper limit value of 100, and sets the alternative value 100 in thesimilarity degree of the face in the image A2.

With the image processing in FIG. 11, if the feature quantity of thepart having the maximum certainty factor is not missing, it is possibleto identify an object image corresponding to the person image 121 basedon the similarity degree of the part.

Also, even if the feature quantity of the part having the maximumcertainty factor is missing, it becomes possible to identify an objectimage corresponding to the person image 121 by setting a pseudosimilarity degree of the part. In this case, it is possible to set asuitable value as the pseudo similarity degree by referencing themagnitude relation of the similarity degree of the second part or lower.

The configurations of the image processing apparatus 101 in FIG. 1 andFIG. 3 are only examples, and a part of the components may be omitted orchanged in accordance with the application and the conditions of theimage processing apparatus 101. For example, if the certainty factorinformation 313 is not used, or another information processing apparatusperforms the certainty factor calculation processing, it is possible toomit the certainty factor calculation unit 301 in FIG. 3. Also, ifanother information processing apparatus generates the featureinformation 312-0 to the feature information 312-N, it is possible toomit the part identification unit 302 and the feature extraction unit303 in FIG. 3.

The images in FIG. 4, FIG. 5, FIG. 7, and FIG. 9 are only examples, andanother image may be used in accordance with the application and theconditions of the image processing apparatus 101. For example, in placeof the two images captured by the two cameras in FIG. 4, two imagescaptured by one camera at two different times may be used. In the samemanner, in place of the plurality of person images captured by the twocameras in FIG. 9, a plurality of person images captured by one cameraat two different times may be used as a sample.

The shape of the area 511 to the area 515 in FIG. 5 may be a shape otherthan a rectangle. The part identification unit 302 may divide the personimage 501 in FIG. 5 into areas of different kinds of or different numberof parts in place of the area 511 to the area 515.

The feature information in FIG. 6, the certainty factor information inFIG. 10, and the similarity degree in FIG. 12 are only examples, andinformation having the other data structures may be used. For example,the start point coordinates (sx, sy) and the end point coordinates (ex,ey) in FIG. 6 may be omitted, and a feature quantity other than RGB maybe used.

The flowcharts in FIG. 2, FIG. 8, and FIG. 11 are only examples, and apart of the processing may omitted or changed in accordance with theconfiguration and conditions of the image processing apparatus 101. Forexample, in step 801 to step 803 in FIG. 8, the certainty factorcalculation unit 301 may use a plurality of person images captured byone camera at two different times as a sample in place of the aplurality of person images captured by the two cameras. In the samemanner, in the image processing in FIG. 11, the object image 311-1captured by the same camera as that used for the person image 121 atdifferent times may be used in place of the object image 311-i capturedby a different camera from that used for the person image 121.

In step 1104 and step 1107 in FIG. 11, the comparison unit 112 maycompare the similarity degree having a predetermined priority of thesecond or lower in place of comparing the similarity degree of the parthaving the first priority. Alternatively, the comparison unit 112 maycompare an addition result produced by adding the similarity degrees ofa plurality of priorities including the first priority, which areweighted in accordance with the certainty factors, respectively.

Expression (1) to Expression (12) are only examples, and an alternativevalue of the similarity degree of a missing part may be calculated bythe other calculation expressions.

FIG. 13 illustrates an example of a configuration of an informationprocessing apparatus used as the image processing apparatus 101 in FIG.1 and FIG. 3. The information processing apparatus in FIG. 13 includes aCPU 1301, a memory 1302, an input device 1303, an output device 1304, anauxiliary storage device 1305, a medium drive unit 1306, and a networkconnection device 1307. These components are coupled with one anotherthrough a bus 1308.

The memory 1302 is a semiconductor memory, for example, a read onlymemory (ROM), a random access memory (RAM), a flash memory, or the like,and stores program and data used for processing. It is possible to usethe memory 1302 as the storage unit 111 in FIG. 1 and FIG. 3.

The CPU 1301 (processor) executes a program using the memory 1302, forexample, so as to operate the comparison unit 112, the certainty factorcalculation unit 301, the part identification unit 302, and the featureextraction unit 303 in FIG. 1 and FIG. 3.

The input device 1303 is, for example, a keyboard, a pointing device, orthe like, and is used for inputting an instruction and information froman operator or a user. The output device 1304 is, for example, a displaydevice, a printer, a speaker, or the like, and is used for outputting aninquiry or an instruction, and a processing result to an operator or auser. It is possible to use the output device 1304 as the output unit113 in FIG. 1 and FIG. 3, and the processing result may be thecomparison result 314.

The auxiliary storage device 1305 is, for example, a magnetic diskdevice, an optical disc device, a magneto-optical disc device, a tapedevice, or the like. The auxiliary storage device 1305 may be a harddisk drive or a flash memory. It is possible for the informationprocessing apparatus to store a program and data in the auxiliarystorage device 1305, and to load the program and the data onto thememory 1302 to use them. It is possible to use the auxiliary storagedevice 1305 as the storage unit 111 in FIG. 1 and FIG. 3.

The medium drive unit 1306 drives the portable recording medium 1309,and accesses its recording contents. The portable recording medium 1309is a memory device, a flexible disk, an optical disc, a magneto-opticaldisc, or the like. The portable recording medium 1309 may be a compactdisk read only memory (CD-ROM), a digital versatile disk (DVD), auniversal serial bus (USB) memory, or the like. It is possible for anoperator or a user to store a program and data into the portablerecording medium 1309, and to load the program and the data into thememory 1302 to use them.

In this manner, the computer-readable recording medium that stores theprogram and the data used for the processing is a physical(non-temporary) recording medium, such as the memory 1302, the auxiliarystorage device 1305, or the portable recording medium 1309.

The network connection device 1307 is coupled to a communicationnetwork, such as a local area network, a wide area network, or the like,and is a communication interface that performs data conversionassociated with communication. The information processing apparatusreceives a program and data from an external device through the networkconnection device 1307, and is capable of using the program and the databy loading them into the memory 1302.

It is possible for the information processing apparatus to receive aprocessing request from a user terminal through the network connectiondevice 1307, to perform image processing, and to transmit the comparisonresult 314 to the user terminal. In this case, it is possible to use thenetwork connection device 1307 as the output unit 113 in FIG. 1 and FIG.3.

In this regard, the information processing apparatus does not have toinclude all the components in FIG. 13, and is capable of omitting a partof the components in accordance with an application and conditions. Forexample, when the information processing apparatus receives a processingrequest from a user terminal through a communication network, the inputdevice 1303 and the output device 1304 may be omitted. Also, if theportable recording medium 1309 or the communication network is not used,the medium drive unit 1306 or the network connection device 1307 may beomitted.

If the information processing apparatus includes a mobile terminalhaving a call feature, such as a smartphone, the information processingapparatus may include a calling device, such as a microphone or aspeaker, and an image capture apparatus, such as a camera.

The detailed descriptions have been given of the disclosed embodimentsand the advantages thereof. It will be obvious to those skilled in theart that various changes, additions, and omissions are possible withoutdeparting from the spirit and scope dearly described in the appendedclaims in the present disclosure.

All examples and conditional language recited herein are intended forpedagogical purposes to aid the reader in understanding the inventionand the concepts contributed by the inventor to furthering the art, andare to be construed as being without limitation to such specificallyrecited examples and conditions, nor does the organization of suchexamples in the specification relate to a showing of the superiority andinferiority of the invention. Although the embodiments of the presentinvention have been described in detail, it should be understood thatthe various changes, substitutions, and alterations could be made heretowithout departing from the spirit and scope of the invention.

What is claimed is:
 1. An image processing apparatus for identifying acandidate area, from among a plurality of candidate areas in a firstimage, corresponding to an object in a second image, the imageprocessing apparatus comprising: a memory; and a processor coupled tothe memory and configured to: divide the plurality of candidate areas inthe first image into a plurality of small candidate areas, respectively,divide an image area of the object in the second image into a pluralityof small areas, perform, regarding each of the plurality of candidateareas, first comparison processing for comparing a small areacorresponding to a first part with a small candidate area correspondingto the first part, the first part being a part of the object, when thereis no a first candidate area lacking image information of the smallcandidate area corresponding to the first part, identify the candidatearea corresponding to the object based on a result of the firstcomparison processing, and when there is the first candidate area,perform second comparison processing for comparing another small areacorresponding to a second part with another small area corresponding tothe second part for each of the plurality of candidate areas, the secondpart being different from the first part, predict missing result on thesmall candidate area corresponding to the first part in the firstcandidate area based on the result of the first comparison processing ona candidate area other than the first candidate area, and a result ofthe second comparison processing on the plurality of candidate areas,and identify the candidate area corresponding to the object based on theresult of the first comparison processing and a result of a prediction.2. The image processing apparatus according to claim 1, wherein theobject is a specified person, and the plurality of candidate areascorrespond to a plurality of persons included in the first image,respectively.
 3. The image processing apparatus according to claim 2,wherein the candidate area having a feature most similar to a feature ofthe image area of the specified person, regarding the first part, isidentified as an area corresponding to the specified person in thesecond image from among the plurality of candidate areas.
 4. The imageprocessing apparatus according to claim 2, wherein the first part is aface, and the second part is a part, other than the face, whose personalfeatures are revealed.
 5. An image processing method of identifying acandidate area, from among a plurality of candidate areas in a firstimage, corresponding to an object in a second image, the imageprocessing method comprising: dividing the plurality of candidate areasin the first image into a plurality of small candidate areas,respectively; dividing an image area of the object in the second imageinto a plurality of small areas; performing, regarding each of theplurality of candidate areas, first comparison processing for comparinga small area corresponding to a first part with a small candidate areacorresponding to the first part, the first part being a part of theobject; when there is no a first candidate area lacking imageinformation of the small candidate area corresponding to the first part,identifying the candidate area corresponding to the object based on aresult of the first comparison processing; and when there is the firstcandidate area, by a processor, performing second comparison processingfor comparing another small area corresponding to a second part withanother small area corresponding to the second part for each of theplurality of candidate areas, the second part being different from thefirst part, predicting missing result on the small candidate areacorresponding to the first part in the first candidate area based on theresult of the first comparison processing on a candidate area other thanthe first candidate area, and a result of the second comparisonprocessing on the plurality of candidate areas, and identifying thecandidate area corresponding to the object based on the result of thefirst comparison processing and a result of a prediction.
 6. The imageprocessing method according to claim 5, wherein the object is aspecified person, and the plurality of candidate areas correspond to aplurality of persons included in the first image, respectively.
 7. Theimage processing method according to claim 6, wherein the candidate areahaving a feature most similar to a feature of the image area of thespecified person, regarding the first part, is identified as an areacorresponding to the specified person in the second image from among theplurality of candidate areas.
 8. The image processing method accordingto claim 6, wherein the first part is a face, and the second part is apart, other than the face, whose personal features are revealed.